Performance Impact of Data Locality in MapReduce on Hadoop

As the foundation for MapReduce processing, Hadoop is one of the fundamental technologies in big data analytics. Hadoop breaks up large data into data blocks, replicates them, and stores them in a distributed storage system. Data blocks can be placed in a machine where the data will be processed (data local), in a machine in the same rack (rack-local), or in a machine in a different rack (off-rack). As the location of a data block gets farther from the processing node, a higher data transfer overhead is incurred. Therefore, the location of a data block can significantly influence the performance of MapReduce processing. The data locality problem has been approached in many ways including, scheduling, data placement, networking, partition/key, and framework. While the majority of the data locality improvement effort is concentrated in the early stages of MapReduce, it is possible to extend it to later stages. These approaches are called Shallow Data Locality (SDL) and Deep Data Locality (DDL) respectively. DDL can be achieved in two ways: pre-arrangement of the data blocks in a way to reduce data movement, and/or micromanipulation of the data within the data blocks. It has been shown that DDL can improve the performance of MapReduce in certain applications. This talk will introduce the concept of Hadoop data locality, its impact on MapReduce performance, and past approaches to improve data locality. The process of DDL, the analytical models and simulation results, and the experimental results with common benchmarking tools will also be presented.